Field of the Invention
The present invention relates generally to computer networks and data distribution and, more specifically, to feeding networks of message brokers with compound data elaborated by dynamic sources.
Description of the Related Art
In the domain of distributed computing, raw data is produced by various data sources and distributed to various client devices via wide area networks (WANs), such as the Internet. As the number of devices, appliances, and applications connected over networks continues to increase, a phenomenon sometimes referred to as the “Internet of Things” (IoT), the amount and types of raw data produced by various data sources within those networks also continues to increase dramatically. Consequently, a problem that oftentimes arises is that the data produced by the data sources is not directly consumable by the client devices. In such cases, the client devices first have to transform the raw data received from the data sources, via techniques such as data filtering, conversion, and derivation, into data that can be consumed by the client devices.
In one example, a data source could produce data in a particular data format that is not recognized by a particular client device, such as when a particular data source transmits messages using a protocol that the client device is unable to interpret. In such a situation, the client device would need to perform a protocol translation on the data from the data source before consuming the data. In another example, a data source could produce temperature data at a particular location once per second. However, a client device could request that the temperature data be delivered once per minute. In such a situation, the client device would need to sample, filter, or aggregate and average the temperature from the data source to yield once-per-minute temperature data. In another example, a data source could produce temperature data at a particular location in degrees Celsius. However, a client device could request that the temperature data be delivered in degrees Fahrenheit. In such a situation, the client device would need to convert the temperature from the data source to yield Fahrenheit temperature data rather than Celsius temperature data. In yet another example, a second data source could produce dew point data at the particular location once per second. A client device that requests relative humidity data would need to access to the temperature and dew point data from both data sources and then derive the relative humidity data the temperature and dew point data from the source data.
In each of these examples, a client application executing on the client device typically performs the necessary data transformation operations on the raw data received from the data source to generate data that can be consumed by the client device. In a typical flow, the client application receives the raw data, performs the needed transformation operations, and then stores the derived data as a new data set in local memory accessible by the client device. The client can then accesses the derived data contained in the new data set directly from the local memory on an as-needed basis.
One drawback of the above approach is that the data produced by a given data source is not transformed and made available to a client device immediately upon being produced by the data source. Another drawback is that the data derived from raw data received from a data source and stored in memory local to a particular client device is not readily accessible by other client devices that also need access to that same derived data. Among other things, these drawbacks can cause substantial latencies between the point in time when raw data from a data source is initially produced and the point in time when data derived from that raw data is actually available to a client device for consumption. Those latencies may result in client devices receiving stale data that is no longer current and/or relevant.
As the foregoing illustrates, what is needed in the art are more effective ways to transform and distribute data to client devices in a distributed computing environment.